Distributed artificial reality system with contextualized hand tracking

ABSTRACT

A system includes an eyewear device configured to present content to a user. A processor is communicatively coupled to the eyewear device. A bracelet device is communicatively coupled to the processor, and includes at least one sensor configured to determine a position signal in response to movement of a user&#39;s hand. A depth camera assembly is communicatively coupled to the processor, and configured to emit a waveform into an environment of the user, and capture a plurality of optical signals from the waveform reflected off of at least one object in the environment. The processor is configured to determine a position of the user&#39;s hand in relation to the environment based in part on the position signal and the plurality of optical signals.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. application Ser.No. 15/919,887, filed Mar. 13, 2018, which is incorporated by referencein its entirety.

BACKGROUND

The present disclosure generally relates to artificial reality systems,and specifically to distributed artificial reality systems with handtracking devices.

Augmented reality (AR) systems and environments allow a user to directlyor indirectly view a real world environment augmented by generatedsensory input, which may be super-imposed on the real world environment.Sensory input can be any form of media, such as sound, video, graphics,etc. Typically, an AR system includes a headset that provides visual andaudio information to the user. Because AR systems allow for users tocontinue to engage with their real world environments in addition to agenerated one, users may have less tolerance for large AR headsetdevices, as opposed to a virtual reality (VR) system in which the useris typically immersed in a fully generated environment. Additionally,smaller form factors facilitate all-day or longer use of artificialreality systems by reducing the friction between a user's experience ofhis or her environment and the artificial reality system itself.

However, the reduced form factor of AR systems produces challenges forproviding sufficient power and computation, and limits the capacity foradding additional features that may enhance the user's AR experience andfacilitate the interaction of the AR system with the environment.Furthermore, hand gestures are an important way in which users interactwith and indicate objects within their environment, but these are notcaptured by a headset device. Because of its limited field of view, aheadset device may be unable to see a user's hands or capture the user'sgestures in response to the simulated environment presented by theheadset. Furthermore, conventional hand tracking systems focus primarilyon simply capturing hand movements, rather than determining what a handmotion means in the context of the user's environment and other signalsdetected by the distributed system. More generally, distributed systemsoften fail to fully integrate different devices and leverage the abilityof a distributed AR system to combine information captured by each ofthe devices in the system.

SUMMARY

A distributed system includes a bracelet device that tracks a user'shand motions with respect to the user's artificial reality environment.The bracelet device is one of several devices in a distributedartificial reality system, which combines sensing, processing and powerstorage across multiple devices. Other devices in the distributed systeminclude an eyewear device and a neckband device. The distributedartificial reality system contextualizes a user's hand motions withinthe user's environment by providing an imaging device that detectsobjects and other features within a user's real-world environment. Theartificial reality may be adjusted in response to the user's handmotion.

A system includes an eyewear device that is configured to presentcontent to a user. A processor is communicatively coupled to the eyeweardevice. A bracelet device is communicatively coupled to the processor.The bracelet device including at least one sensor configured todetermine a position signal in response to movement of the user's hand.A depth camera assembly is communicatively coupled to the processor. Thedepth camera assembly is configured to emit a waveform into anenvironment of the user and capture a plurality of optical signals fromthe waveform reflected off of at least one object in the environment.The processor is configured to determine a position of the user's handin relation to the environment based in part on the position signal andthe plurality of optical signals.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a distributed artificial reality system, inaccordance with one or more embodiments.

FIG. 2 is a diagram of a distributed artificial reality system worn by auser, in accordance with one or more embodiments.

FIG. 3 is a diagram of an imaging device of a distributed artificialreality system in a local environment, in accordance with one or moreembodiments.

FIG. 4 is a diagram of signal communications in a distributed artificialreality system, in accordance with one or more embodiments.

FIG. 5 is a block diagram of a distributed artificial reality system, inaccordance with one or more embodiments.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

AR and/or mixed reality (MR) devices allow a user to directly orindirectly view a real world environment augmented by generated sensoryinput, such as sound, video, graphics, etc. The generated sensory inputmay be super-imposed on the real world environment, allowing the user tointeract with both simultaneously. To allow the user to continue to viewand interact with his or her real world environment while alsoperceiving the augmented environment, AR devices ideally provide aminimally invasive interface for the user that can be easily worn forlong periods of time without interfering with the user's interactionswith the environment. One category of AR, virtual reality (VR), and/orMR device designs includes a near-eye display (NED) with opticalelements that provide the user with generated visual input such asgraphics or video. A NED may be imbedded in an eyewear device in theform of eyeglasses, which rest on the bridge of a user's nose. However,to accommodate the reduced form factor of the NED as a pair ofeyeglasses, computation, battery, and additional functions are moved offof the NED and onto separate linked devices. The result is a distributedAR system of multiple independent devices that together provide a fullAR experience for the user.

Gestures and hand motions are important ways in which people interactwith their real world environments. To determine how a user responds toan artificial reality, a distributed AR system can capture handmovements and translate them into gestures within and responsive to thegenerated artificial reality. Hand tracking systems, however, generallyfocus simply on detecting user hand motions, and don't necessarilycombine these signals with the functionality of other devices within thedistributed AR system. For example, a pointing gesture may have manypossible meanings depending on the environment of the user; if there isan object in the real world, a user pointing to the object has a verydifferent meaning than simply capturing that the user is making apointing gesture. Thus, determining gestures with respect to a user'sreal world environment is important to understanding how a user isinteracting with the artificial reality that might be overlaid on top ofit.

The present disclosure provides small form factor devices to presentvisual and audio content to a user and also track user hand gestures bydistributing functions across several devices. The resulting distributedartificial reality system allows for hand tracking that detects a user'sgestures with respect to the artificial and real world environment,providing context for the user's gestures not necessarily achievable bya hand tracking system alone. Thus, the distributed AR system leveragessignals collected across multiple devices to provide a more immersiveartificial reality system that better responds to a user's interactionwith the artificial reality.

Embodiments of the present disclosure may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a head-mounted display (HMD) connectedto a host computer system, a standalone HMD, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewer.

System Overview

FIG. 1 is a diagram of a distributed artificial reality system 100, inaccordance with one or more embodiments. The distributed system 100includes a bracelet 105, an eyewear device 110, and a neckband 115. Inalternate embodiments, the distributed system 100 may include additionalcomponents (e.g., a mobile device as discussed in detail below withregard to FIGS. 4 and 5).

Bracelet

The bracelet 105 detects movement of a user's hand. The bracelet 105includes a position sensor 120, a camera 125 and magnetic sensors 130.The bracelet 105 is shown as a circlet formed from a band with roundededges. The bracelet 105 fits around the wrist of a user, which is shownin more detail with respect to FIG. 2. The bracelet 105 may be formedfrom any flexible material, such as an elastomer or other polymermaterial. The sizing of the bracelet 105 may be adjustable, such thatthe band size can be fit to the wrist of each individual user.

As shown, the bracelet 105 includes a position sensor 120. There may beany number of position sensors 120 located at various points on thebracelet 105. The one or more position sensors may be located externalto an inertial measurement unit (IMU) (not shown), internal to an IMU,or some combination thereof. The position sensor 120 may be any sensorcapable of determining a position of the bracelet 105 and generating asignal in response to movement of the bracelet 105. Since the bracelet105 is worn on a user's wrist, the position sensors 120 thereforeprovide position signals that result from the movement of a user's arm.In some embodiments, the position sensor 120 tracks the position of thebracelet 105 over time, using a previous location data point todetermine subsequent positions. In some embodiments, the position sensor120 may be an accelerometer that measures translational motion (e.g.,forward/back, up/down, left/right). Additionally or alternatively, theposition sensor 120 is a gyroscope that measures rotational motion(e.g., pitch, yaw, and roll). In some embodiments, the multipleaccelerometers and multiple gyroscopes located on the bracelet 105together provide position signals indicating movement of the bracelet105 in six degrees of freedom. The position sensor 120 may be a MEMSdevice.

The one or more position sensors 120 provide position signals to an IMU(not shown) which may be located on the bracelet 105, eyewear device110, and/or neckband 115. The IMU uses the position signals from the oneor more position sensors 120 to estimate a position of the bracelet 105.The IMU may estimate a relative position of the bracelet 105 withrespect to the eyewear device 110, neckband 115, and/or any other devicein a distributed artificial reality system 100, such as a mobile deviceas described FIGS. 4-5. In some embodiments, the IMU rapidly samples themeasurement signals from the one or more position sensors 120 andcalculates the estimated position of the bracelet 105 from the sampleddata. For example, the IMU integrates the measurement signals receivedfrom the one or more position sensors 120 over time to estimate avelocity vector, and integrates the velocity vector over time todetermine an estimated position of a reference point of the bracelet105. Some or all of the computation for the IMU processing of theposition signals from the one or more position sensors 120 may becarried out by the computation compartment 170 of the neckband 115, theeyewear device 110, a mobile device (not shown), or any other device inthe distributed artificial reality system 100.

An IMU, computation compartment 170, or any other processor thatdetermines position from the position sensor 120 may receive one or morecalibration parameters from the bracelet 105. The one or morecalibration parameters are used to maintain tracking of the bracelet105. Based on a received calibration parameter, an IMU may adjust one ormore IMU parameters (e.g., sample rate). The adjustment may bedetermined by the computation compartment 170 of the neckband 115, aprocessor of a mobile device, or any other processor of a device in thedistributed artificial reality system 100. In some embodiments, certaincalibration parameters cause the IMU to update an initial position ofthe reference point so it corresponds to a next calibrated position ofthe reference point. Updating the initial position of the referencepoint at the next calibrated position of the reference point helpsreduce accumulated error associated with the determined estimatedposition of the bracelet 105. The accumulated error, also referred to asdrift error, causes the estimated position of the reference point to“drift” away from the actual position of the reference point over time.In some examples, the IMU receives position information from both theone or more position sensors 120 on the bracelet 105 as well as positionsensors (not shown) on the eyewear device 110 and position sensors (notshown) on the neckband 115.

As shown in FIG. 1, the bracelet 105 includes a camera 125. There may beone or more cameras 125 located on the bracelet 105. The one or morecameras 125 capture gesture information of a user's hand, as visiblefrom the bracelet 105. Gesture information about the user's handincludes finger motions and pose, as well as palm motions pose. Agesture, pose, position or movement of a user's hand may be pointing,waving, or any other signal made by a user's hand. While the visualinformation may be captured by the one or more cameras 125, a handgesture, pose or position of the user's hand may be determined from thevisual information by a processor not physically located in the bracelet105. The processor may be located on the neckband 115, eyewear device110, or any other device in the distributed artificial reality system100, such as a mobile device as described with reference to FIGS. 4-5.The visual information captured by the one or more cameras 125 may beused in a kinematic model of the hand, which relates visual detectedmovements of the user's hand to a hand gesture. The kinematic model maybe machine learned, such that over time a model relating the movementinformation captured by the one or more cameras 125 is adapted to auser's hand gestures. The kinematic model may combine visual informationcollected from the one or more cameras 125 with other information fromother sensors on the bracelet 105, eyewear device 110 and/or neckband115. For example, the visual information from the camera 125 may becombined with movement information determined by the position sensor 120and/or IMU. In other examples, the visual information from the camera125 may be combined with visual information obtained from a cameralocated on the eyewear device 110 and/or neckband 115 (not shown). Thekinematic model may be stored in a controller, or other software module(not shown). For example, the kinematic model may be stored in a mobiledevice, such as the mobile device described with reference to FIGS. 4-5.The processor communicates with the controller or software model anduses the kinematic model to determine a pose, gesture or hand signalfrom any of the signals from the camera 125, position sensor 120,magnetic sensors 130, or any other sensors located on the bracelet 105.The processor is located on the neckband 115, eyewear device 110, or anyother device in the distributed artificial reality system 100, such as amobile device as described with reference to FIGS. 4-5.

In some examples, the camera may be a depth camera assembly thatcollects depth image data of a user's hand. A hand tracking unit,located on any of the eyewear device 110, neckband 115, and/or a mobiledevice may include a neural network that uses the depth image data todetermine a hand pose of the user's hand. Additional details regardingdetermining a pose, gesture, or other information about a user's handmay be found at, e.g., U.S. patent application Ser. No. 15/487,361,which is hereby incorporated by reference in its entirety.

The signals describing the movement of a user's hand as determined bythe camera 125, one or more position sensors 120 and IMU are combinedwith the signals produced by the magnetic sensors 130 which measure arelative position of the bracelet 105 with respect to a backgroundmagnetic field. The magnetic sensors 130 may be located anywhere on thebracelet 105, and there may be any number of magnetic sensors 130distributed on the bracelet 105. The magnetic sensors 130 may bemagnetometers, MEMS magnetic field sensors, or any other magnetic sensorcapable of detecting the direction, strength, and/or change in amagnetic field.

A baseline, background magnetic field exists in any environment in whichthe bracelet 105 operates, whether this be the result of the Earth'smagnetic field or from the presence of other electronic devices in theenvironment that generate electromagnetic radiation. Regardless of thesource, an ambient, background magnetic field exists in the environment,though it may have different directions and/or magnitudes. The magneticsensors 130 measure a vector direction of the background magnetic field.When the bracelet 105 moves in response to movement of a user's arm, thevector direction measured by the magnetic sensors 130 changes inresponse to the new orientation of the bracelet 105 with respect to thebackground magnetic field. In some embodiments, it is assumed that thebackground magnetic field does not change in time, and thus that thedetected changes in the magnetic field result from the movement of thebracelet 105. In some embodiments, the magnetic sensors 130 are 3Dmagnetometers and thus each detect changes in three degrees of freedom.In other embodiments, each magnetic sensor may be configured to generatea signal in response to changes in a single direction, so that toachieve three degrees of freedom, three different magnetic sensors 130are configured to detect along three separate axes.

Together, the magnetic sensors 130 produce signals giving the gradientof the change in the measured magnetic field resulting from the bracelet105 movement with respect to a stationary, background magnetic field.From this gradient, a velocity of the bracelet 105 may be determined,which may contain both rotational and linear velocity components.However, rotational velocity may be independently measured by theposition sensors 120, such as by gyroscope position sensors 120. Thus,combining the measurement from the magnetic sensors 130 with the knownrotational measurement from the position sensors 120 allows for thelinear velocity component of the gradient to be independently determinedfrom the measured gradient. This linear velocity may thus be determinedwithout being subject to error from drift in the integrated linearaccelerometer and IMU measurement that may affect the accuracy of theposition sensors 120. Similarly, the rotational component of velocity ofthe bracelet 105 may be determined from the gradient measured by themagnetic sensors 130 if the known linear velocity measured byaccelerometer position sensors 120 is used to determine an unknownrotational velocity.

In some embodiments, measurements from any of the position sensor 120,camera 125 and magnetic sensors 130 may be combined into a kinematicmodel to determine a hand gesture, the position of the user's arm,and/or the position of the user's forearm. In some embodiments,measurements from any of the position sensor 120, camera 125, andmagnetic sensors 130 may be used in a Simultaneous Localization andMapping (SLAM) calculation, which may be carried out by the computationcompartment 170 located on the neckband 115 and/or any other device inthe distributed artificial reality system 100, such as a mobile deviceas described in FIGS. 4-5.

In some embodiments, the bracelet 105 includes a wireless gateway ordirectional antenna (not shown), located anywhere on the bracelet 105,through which it transmits and receives information from the eyeweardevice 110, neckband 115, and/or any other device in the distributedartificial reality system 100. The wireless connection between thebracelet 105, eyewear device 110 and/or neckband 115 may be a WiFiconnection, a Bluetooth connection, or any other wireless connectioncapable of transmitting and receiving information. The wireless gatewaymay also connect the bracelet 105 to a mobile device, as described infurther detail with reference to FIGS. 4-5. In other embodiments, thebracelet 105 is connected to the neckband 115, eyewear device 110, orany other device in a distributed artificial reality system, such as amobile device, through a wired connection. In these examples, a wiredconnection may provide power to the bracelet 105 and/or transmitinformation from the bracelet 105 to a second device. For example, thebattery compartments 155 in the neckband 115 may provide power to thebracelet 105 through a wired connection. The connecting wire may beretractable or otherwise adjustable in length.

In some embodiments, the bracelet 105 includes a battery compartment(not shown) through which any of the position sensor 120, camera 125 andmagnetic sensors 130 are powered. The power source in the batterycompartment may be re-chargeable. The power source in the batterycompartment may be lithium ion batteries, lithium-polymer battery,primary lithium batteries, alkaline batteries, or any other form ofpower storage.

Eyewear Device

The eyewear device 110 provides content to a user of the distributedsystem 100. The eyewear device 110 includes two optical systems 135. Theeyewear device 110 may also include a variety of sensors, such as one ormore passive sensors, one or more active sensors, one or more audiodevices, an eye tracker system, an IMU (not shown), or some combinationthereof. As shown in FIG. 1, the eyewear device 110 and optical systems135 are formed in the shape of eyeglasses, with the two optical systems135 acting as eyeglass “lenses” within a frame 140. The frame 140 isattached to a neckband 115 by a connector 145, which connects to theneckband 115 by a connector junction 150.

Optical systems 135 present visual media to a user. Each of the opticalsystems 135 may include a display assembly. In some embodiments, whenthe eyewear device 110 is configured as an AR eyewear device, thedisplay assembly also allows and/or directs light from a local areasurrounding the eyewear device 110 to an eyebox (i.e., a region in spacethat would be occupied by a user's eye). The optical systems 135 mayinclude corrective lenses, which may be customizable for a user'seyeglasses prescription. The optical systems 135 may be bifocalcorrective lenses. The optical systems 135 may be trifocal correctivelenses.

The display assembly is used to present visual media to a user bygenerating image light with a projector and conveying the output lightto a user's eye in an eyebox through any number of waveguides, gratings,light expansions, etc. The display assembly thus provides contentthrough generated image light, which may be overlaid on top of a user'sreal-world environment. The display assembly may be composed of one ormore materials (e.g., plastic, glass, etc.) with one or more refractiveindices that effectively minimize the weight and widen a field of viewof the eyewear device 110 visual system. In alternate configurations,the eyewear device 110 includes one or more elements between the displayassembly and the eye. The elements may act to, e.g., correct aberrationsin image light emitted from the display assembly, correct aberrationsfor any light source due to the user's visual prescription needs,magnify image light, perform some other optical adjustment of imagelight emitted from the display assembly, or some combination thereof. Anelement may include an aperture, a Fresnel lens, a convex lens, aconcave lens, a liquid crystal lens, a liquid or other deformablesurface lens, a diffractive element, a waveguide, a filter, a polarizer,a diffuser, a fiber taper, one or more reflective surfaces, a polarizingreflective surface, a birefringent element, or any other suitableoptical element that affects image light emitted from the displayassembly. Additional details describing an example of the displayassembly may be found at, e.g., U.S. patent application Ser. No.15/704,190, which is hereby incorporated by reference in its entirety.

Examples of media presented by the eyewear device 110 include one ormore images, text, video, audio, or some combination thereof. Theeyewear device 110 can be configured to operate, in the visual domain,as a VR NED, an AR NED, a mixed reality (MR) NED, or some combinationthereof. For example, in some embodiments, the eyewear device 110 mayaugment views of a physical, real-world environment withcomputer-generated elements (e.g., images, video, sound, etc.). Theeyewear device 110 may include a speaker or any other means of conveyingaudio to a user, such as bone conduction, cartilage conduction, etc.

The media presented by the eyewear device 110 may be adjusted inresponse to a user's hand gesture as detected by the bracelet 105. Forexample, the bracelet 105 may determine that a user is pointing to avirtual object in the visual artificial reality displayed in the eyeweardevice 110. In response, the view shown by the eyewear device 110 mayzoom in on the object, display information about the object, bring theobject into focus, or any other adjustment that responds to the user'sgesture.

In other embodiments, the eyewear device 110 does not present media orinformation to a user. For example, the eyewear device 110 may be usedin conjunction with a separate display, such as a coupled mobile deviceor laptop (not shown). In other embodiments, the eyewear device 110 maybe used for various research purposes, training applications, biometricsapplications (e.g., fatigue or stress detection), automotiveapplications, communications systems for the disabled, or any otherapplication in which eye tracking or other sensing functions can beused.

The eyewear device 110 may include embedded sensors (not shown) such as1-dimensional (1D), 2-dimensional (2D) imagers, or scanners forlocalization and stabilization of the eyewear device 110, as well assensors for understanding the user's intent and attention through time.The sensors located on the eyewear device 110 may be used for SLAMcalculations, which may be carried out in whole or in part by theprocessor embedded in the computation compartment 170 and/or a processorlocated in a coupled mobile device, as described in further detail withreference to FIGS. 4-5. Embedded sensors located on the eyewear device110 may have associated processing and computation capabilities.

In some embodiments, the eyewear device 110 further includes an eyetracking system (not shown) for tracking a position of one or both eyesof a user. Note that information about the position of the eye alsoincludes information about an orientation of the eye, i.e., informationabout user's eye-gaze. Based on the determined and tracked position andorientation of the eye, the eyewear device 110 adjusts image lightemitted from one or both of the display assemblies. In some embodiments,the eyewear device 110 adjusts focus of the image light through theoptical systems 135 and ensures that the image light is in focus at thedetermined angle of eye-gaze in order to mitigate thevergence-accommodation conflict (VAC). Additionally or alternatively,the eyewear device 110 adjusts resolution of the image light byperforming foveated rendering of the image light, based on the positionof the eye. Additionally or alternatively, the eyewear device 110 usesthe information on a gaze position and orientation to provide contextualawareness for the user's attention, whether on real or virtual content.The eye tracker generally includes an illumination source and an imagingdevice (camera). In some embodiments, components of the eye tracker areintegrated into the display assembly. In alternate embodiments,components of the eye tracker are integrated into the frame 140.Additional details regarding incorporation of eye tracking system andeyewear devices may be found at, e.g., U.S. patent application Ser. No.15/644,203, which is hereby incorporated by reference in its entirety.

Computation for the eye-tracking system may be carried out by theprocessor located in the computation compartment 170 and/or a coupledmobile device, as described in further detail with reference to FIG.4-5. The eyewear device 110 may include an IMU sensor (not shown) todetermine the position of the eyewear device relative to a user'senvironment, as well as detect user movement. The IMU sensor may alsodetermine the relative spatial relationship between the eyewear device110 and the neckband 115, and/or the spatial relationship between theeyewear device 110 and the bracelet 105, which may provide informationabout the position of the user's head relative to the position of theuser's body and hand. Here the neckband 115 may also include an IMUsensor (not shown) to facilitate alignment and orientation of theneckband 115 relative to the eyewear device 110. The IMU sensor on theneckband 115 may determine the orientation of the neckband 115 when itoperates independently of the eyewear device 110. The eyewear device 110may also include a depth camera assembly (DCA) (not shown), which may bea Time-of-Flight (TOF) camera, a Structured Light (SL) camera, a passiveand/or active stereo system, and may include an infrared (IR) lightsource and detection camera. A DCA captures visual information ofvarious depth zones of a scene of an area surrounding a device, such asthe eyewear device 110. Additional details describing the DCA may befound at, e.g., U.S. patent application Ser. No. 15/372,779, which ishereby incorporated by reference in its entirety. The eyewear device 110may include a variety of passive sensors, such as a red green and blue(RGB) color camera, passive locator sensors, etc. Cameras on the eyeweardevice 110 may supplement visual information about the user's hand, asdetermined by the one or more cameras 125 located on the bracelet 105.The eyewear device 110 may include a variety of active sensors, such asstructured light sensors, active locators, etc. The number of activesensors may be minimized to reduce overall weight, power consumption andheat generation on the eyewear device 110. Active and passive sensors,as well as camera systems may be placed anywhere on the eyewear device110.

Neckband

The neckband 115 is a wearable device that performs processing intensiveoperations offloaded to it from other devices (e.g., the bracelet 105,the eyewear device 110, a mobile device, etc.). As shown in FIG. 1, theneckband 115 is connected to the eyewear device 110 by the connectionjunction 150. The neckband 115 is wirelessly connected to the bracelet105 through any standard wireless signal connection. The neckband 115includes battery compartments 155 located on each of the first arm 160,second arm 165 and computation compartment 170. As shown, batterycompartment 155 is embedded in both the first arm 160 and the second arm165. As shown, a computation compartment 170 is connected to both thefirst arm 160 and the second arm 165.

Any of the above components may be located in any other part of theneckband 115. There may be any number of battery compartments 155distributed across the neckband 115. There may be any number ofcomputation compartments 170 distributed across the neckband 115. Thebattery compartment 155 and computation compartment 170 are preferablylocated such that weight is distributed evenly across the neckband 115,from left-to-right across the device symmetrical axis. Batterycompartments may be located symmetrically across the device symmetricalaxis, or may be asymmetrically distributed to balance out the weight ofother sensors or compartments. For example, the battery compartments 155may be located on the first arm 160 to balance out a computationcompartment located on the second arm 165.

The neckband 115, composed of the first arm 160, the second arm 165 andthe computation compartment 170, is formed in a “U” shape that conformsto the user's neck. The neckband 115 is worn around a user's neck, whilethe eyewear device 110 is worn on the user's head (not shown) asdescribed in further detail with respect to FIG. 2. The first arm 160and second arm 165 of the neckband 115 may each rest on the top of auser's shoulders close to his or her neck such that the weight of thefirst arm 160 and second arm 165 are carried by the user's neck base andshoulders. The computation compartment 170 may sit on the back of auser's neck. The connector 145 is long enough to allow the eyeweardevice 110 to be worn on a user's head while the neckband 115 restsaround the user's neck. The connector 145 may be adjustable, allowingeach user to customize the length of connector 145.

The power source in the battery compartment 155 may be in one or moresub-assemblies (with two as shown) where the unit embedded in the firstarm 160 and/or the battery compartment 155 in the second arm 165 powersthe eyewear device 110. Each battery compartment 155 houses a powersource (e.g., batteries) which may be re-chargeable. The power source inthe battery compartment 155 may be lithium ion batteries,lithium-polymer battery, primary lithium batteries, alkaline batteries,or any other form of power storage. If more than one batterysub-assembly is utilized, the battery compartment 155 of the first arm160 may have a different battery or power capacity than the battery inthe battery compartment 155 of the second arm 165. The computationcompartment 170 may have its own power source (not shown) and/or may bepowered by a power source in the battery compartments 155. Locating thebattery compartments 155 on the neckband 115 distributes the weight andheat generated by the battery compartment 155 from the eyewear device110 to the neckband, which may better diffuse and disperse heat, andalso utilizes the carrying capacity of a user's neck base and shoulders.Locating the battery compartments 155, computation compartment 170 andany number of other sensors on the neckband 115 may also better regulatethe heat exposure of each of these elements, as positioning them next toa user's neck may protect them from solar and environmental heat sources

The computation compartment 170 houses a processor (not shown), whichprocesses information generated by any of the sensors or camera systemson the bracelet 105, eyewear device 110 and/or the neckband 115. Theprocessor of the computation compartment 170 is communicatively coupledto the eyewear device 110. The processor of the computation compartment170 is communicatively coupled to the bracelet 105. The communicationbetween the bracelet 105, eyewear device 110 and neckband 115 isdescribed in further detail with reference to FIG. 3. The communicationbetween the processor and either of the bracelet 105 and eyewear device110 may be through any of the signals described with reference to FIG.3. Information generated by the bracelet 105, eyewear device 110 and theneckband 115 may also be processed by a mobile device, such as themobile device described in further detail with reference to FIGS. 4-5. Aprocessor of a mobile device, such as the mobile device described infurther detail with reference to FIGS. 4-5, may be communicativelycoupled to the eyewear device 110 and/or the bracelet 105. The processorin the computation compartment 170 may process information generated byeach of the bracelet 105, the eyewear device 110 and the neckband 115.The connector 145 conveys information between the eyewear device 110 andthe neckband 115, and between the eyewear device 110 and the processorin the computation compartment 170. The bracelet 105 may conveyinformation to the computation compartment 170 via a wireless gatewaylocated on the bracelet 105 and neckband 115. The wireless gateway ofthe neckband 115 is described in further detail with reference to FIG.5. In some examples, the first arm 160, and second arm 165 may also eachhave an embedded processor (not shown). In these examples, the connector145 conveys information between the eyewear device 110 and the processorin each of the first arm 160, the second arm 165 and the computationcompartment 170. In these examples, information received from thebracelet 105 through a wireless gateway may be distributed to each ofthe first arm 160, second arm 165 and the computation compartment 170for processing. The information may be in the form of optical data,electrical data, or any other transmittable data form. Moving theprocessing of information generated by the bracelet 105 and eyeweardevice 110 to the neckband 115 reduces the weight and heat generation ofthe bracelet 105 and eyewear device 110, making them more comfortable tothe user and encouraging user over longer time scales.

The processor embedded in the computation compartment 170 may computeall inertial and spatial calculations from IMU sensors located on thebracelet 105 and eyewear device 110. The processor embedded in thecomputation compartment 170 may compute all calculations from the activesensors, passive sensors, and camera systems located on the eyeweardevice 110, as well as the position sensor 120, camera 125 and magneticsensors 130 on the bracelet 105. The processor embedded in thecomputation compartment 170 may perform all computations frominformation provided by any sensor located on the bracelet 105 and/oreyewear device 110. The processor embedded in the computationcompartment 170 may perform all computation from information provided byany sensor located on the bracelet 105 and/or eyewear device 110 inconjunction with a processor located on a coupled external device, suchas a mobile device as described in further detail with reference toFIGS. 4-5.

The neckband 115 may include a multifunction compartment (not shown).The multifunction compartment may be a customizable compartment in whichadditional feature units may be inserted and removed by a user.Additional features may be selected and customized by the user uponpurchase of the neckband 115. Examples of units that may be included ina multifunction compartment are: an imaging device, a memory unit, aprocessing unit, a microphone array, a projector, a camera, etc. Thesemay be described in further detail with reference to FIG. 5.

The computation compartment 170 and battery compartment 155 may besegments of the neckband 115 as shown in FIG. 1. However, computationcompartments 170 and battery compartments 155 may also be anysub-structures of neckband 115, such as compartments embedded withinneckband 115, compartments coupled to sensors embedded in neckband 115,and/or may be located anywhere on neckband 115.

In some examples, the connector 145 conveys information from the eyeweardevice 110 to the neckband 115. Sensors located on the eyewear device110 may provide the processor embedded in the computation compartment170 with sensing data, which may be processed by the processor in thecomputation compartment 170. The computation compartment 170 may conveythe results of its computation to the eyewear device 110. For example,if the result of the processor in the computation compartment 170 is arendered result to be displayed to a user, the computation compartmentsends the information through the connector 145 to be displayed on theoptical systems 135. In some examples, there may be multiple connectors145. For example, one connector 145 may convey power, while anotherconnector 145 may convey information.

In some examples, the connector 145 provides power to the eyewear device110 through magnetic induction at the connector junctions (not shown)with the frame 140. In this example, the connector 145 may be coupled tothe frame 140 with retention magnets. In other examples, the connector145 provides power from the neckband 115 to the eyewear device 110through any conventional power coupling technique. The connector 145 isflexible to allow for independent movement of the eyewear device 110relative to the neckband 115. The connector 145 may be retractable, orotherwise adjustable to provide the correct length between thenear-eye-display and the neckband 115 for each user, since the distancebetween a user's head and neck may vary.

In some examples, the eyewear device 110 is wirelessly coupled with theneckband 115. In these examples, the processor embedded in thecomputation compartment 170 receives information from the eyewear device110 and the sensors and camera assemblies located on the eyewear device110 through the wireless signal connection, and may transmit informationback to the eyewear device 110 through the wireless signal connection.The wireless connection between the eyewear device 110 and the neckband115 may be through a wireless gateway (not shown) or directionalantenna, located in the first arm 160 and/or second arm 165 and/or onthe eyewear device 110. The wireless connection between the eyeweardevice 110 and the neckband 115 may be a WiFi connection, a Bluetoothconnection, or any other wireless connection capable of transmitting andreceiving information. The wireless gateway may also connect the eyeweardevice 102 and/or the neckband 115 to a mobile device, as described infurther detail with reference to FIGS. 4-5.

In some examples in which the eyewear device 110 is wirelessly coupledwith the neckband 115, the connector 145 may only transmit power betweenthe neckband 115 and the eyewear device 110. Information between theeyewear device 110 and neckband 115 would thus be transmittedwirelessly. In these examples, the connector 145 may be thinner. In someexamples in which the eyewear device 110 is wirelessly coupled with theneckband 115, power may be transmitted between the eyewear device 110and the neckband 115 via wireless power induction. Additionally oralternatively, power may be transmitted between the neckband 115 and thebracelet 105 via wireless power induction. In some examples, there maybe a separate battery or power source located in the eyewear device 110.In some examples in which the eyewear device 110 is wirelessly coupledwith the neckband 115, the addition of a connector 145 may be optional.

Interaction Between Devices in Distributed System

The visual component of the artificial reality generated by the opticalsystems 135 on the eyewear device 110 or the neckband 115 may change inresponse to movement of the user's hand as measured by the bracelet 105.Additionally or alternatively, audio components of an artificialreality, produced by either the eyewear device 110 or the neckband 115may be adjusted in response to movements of the user's hand as measuredby the bracelet 105. Any other component of the artificial realitygenerated by either the eyewear device 110 and/or the neckband 115 maybe altered in response to a user's gestures or hand movement as measuredby the bracelet 105. For example, the bracelet 105 may determine that auser is pointing to a virtual object that is displayed in the opticalsystems 135, and in response, the optical system 135 may zoom in on theobject.

The bracelet 105, eyewear device 110 and neckband 115 architecture thusallows the user's artificial reality experience to be through small formfactor devices, while still maintaining the processing and battery powernecessary to provide a full artificial reality experience. Furthermore,the bracelet 105 allows the distributed artificial reality system 100 todetermine a user's hand motions in response to the artificial realitydisplayed and generated through the eyewear device 110 and neckband 115.The form factor of the bracelet 105 can be reduced since its processingmay be carried out by the neckband 115. The movement detected by thebracelet 105 can be further contextualized by combining user handgestures with information about the user's real world environment, asdetermined by sensors located on the eyewear device 110 and/or neckband115. This is described in further detail with respect to FIG. 3.

Distributed Artificial Reality System and a User

FIG. 2 is a diagram of a distributed artificial reality system beingworn by a user, in accordance with one or more embodiments. The user 200wears the bracelet 105 close to his or her hand 205. The bracelet 105conforms to the shape of the user 200's wrist. The user 200 wears theeyewear device 102 on his or her head like a pair of eyeglasses. Theneckband 115 is shown around the user neck 210, such that thecomputation compartment 130 is on the back of the user neck 210 and thesecond arm 165 rests on the side of the user neck 210. Thus, the weightof the neckband 115 is distributed to the user's shoulders, and theneckband 115 is conformed to the shape of the user neck 210.

The bracelet 105 conforms to the user's wrist so that movement of theuser's hand 205 and/or arm (not shown) do not result in shifting of thesensors on the bracelet 105 with respect to the hand 205. The camera 125may be secured so that the user's hand 205 is in line of sight of thecamera 125. The magnetic sensors (not shown), combined with the positionsensor 120 and camera 125 may collectively measure movements of theuser's hand 205 and convey the measurements to the computationcompartment 170 of the neckband 115. Since the user may tolerate aheavier load on their shoulders, weight of components that mightotherwise be located on the bracelet 105 and/or eyewear device 110 maybe shifted to the neckband 115, facilitating freer movement of theuser's hand 205.

Hand Tracking in the Context of a Local Environment

FIG. 3 is a diagram of an imaging device 310 of a distributed artificialreality system in a local environment 305, in accordance with one ormore embodiments. The imaging device 310 may be a DCA, which collectsinformation about a user's local environment 305. The imagining device310 may be located on either the eyewear device 110 and/or the neckband115. The imaging device 310 includes one or more sources and one or morelight sensors (e.g., a camera, a video camera). The light sourceilluminates the local environment, which allows the imagining device 310to detect the presence of objects (e.g., objects 320 and 325, a user'shands, etc.) in the local environment. The bracelet 105 tracks a user'shand and determines a hand gesture 330 with respect to the localenvironment 305 and the detected objects 320 and 325. As referred toherein, a hand gesture 330 may also be referred to as a pose of a user'shand, a position of a user's hand, or a hand movement.

The imaging device 310 emits a waveform 315 into the local environment305. The wavelength of the waveform 315 may be in a visible band (˜380nm to 750 nm), an infrared (IR) band (˜750 nm to 1500 nm), anultraviolet band (10 nm to 380 nm), another portion of theelectromagnetic spectrum, or some combination thereof. The light sensorsmay be sensitive to the same range of wavelengths emitted as thewaveform 315. The light sensors may also be sensitive to light havingvisible wavelengths as well as the light emitted as the waveform 315.For example, the light sensors may be red, green, blue, IR (RGBI)cameras. In some embodiments, the light sensors may be a camera inaddition to a complementary metal-oxide-semiconductor (CMOS) imager,other light sensitive device, or some combination thereof.

The light sensors of the imaging device 310 detect the presence ofobjects (e.g., the objects 320 and 325, the user's hands, etc.) in thelocal environment 305. For example, the waveform 315 emitted into thelocal environment 305 reflects off of the objects 320 and 325. Inaddition to the reflected waveform 315, objects 320 and 325 reflectincident ambient light that may be collected by the light sensors of theimaging device 310. The reflected ambient and waveform 315 light iscollected by the light sensors and compared to the emitted waveform 315to determine characteristics of objects 320 and 325 such as the distanceof objects 320 and 325 from the imaging device 310, the shape of theobjects 320 and 325, the surface texture of the objects 320 and 325,etc. In some examples, the waveform 315 is structured light (SL), whichmay be in any SL pattern, such as a symmetric or quasi dot pattern,grid, horizontal bars, etc. The imaging device 310 compares the emittedSL to the structure of the reflected light to detect the characteristicsof objects 320 and 325. In some embodiments, the imaging device 310 mayuse Time of Flight (ToF) depth determination techniques in which, e.g.,the characteristics of objects 320 and 325 are determined from a timedelay between the emitted waveform 315 and a detected reflection of aportion of the waveform 315 off of the objects 320 and 325. In someembodiments, the imaging device 310 may use stereo imaging techniques,in which the characteristics of objects 320 and 325 are determinedthrough a stereo image. Additionally, in some embodiments, one or moredepth determination techniques may be combined to determine depthinformation for objects (e.g., the object 320, the object 325, a user'shands, etc.) in the local environment 305. Any other depth sensing maybe used by the imaging device 310 to determine the characteristics ofobjects 320 and 325 and/or a hand of the user. In some embodiments, theimaging device 310 does an initial scan of waveform 315 in the localenvironment 305 to detect objects 320 and 325, and then selectivelyemits a second waveform 315 localized around the detected objects 320and 325.

The imaging device 310 conveys the collected information about objects320 and 325 and conveys them to a processor of a distributed artificialreality system, such as a processor in the computation compartment 170of the neckband 115, the processor of a mobile device, such as mobiledevice 405 or 534 as shown in FIGS. 4-5, or any other processor in anyother device in the distributed artificial reality system. The processoralso receives a number of visual signals from the camera 125, positionsignals from the position sensor 120, orientation signals from themagnetic sensors 130, or any other signals from a bracelet 105 thattogether may indicate a hand gesture 330. The processor determines thehand gesture 330 from any of the signals received from the bracelet 105.The imaging device 310 is thus communicatively coupled to a processor ina distributed artificial reality system, which may be through any wiredor wireless signal.

The processor combines information about the objects located in thelocal environment 305 as determined by the imaging device 310 with thehand gesture 330. The hand gesture 330 is determined by the processorfrom sensor information conveyed to the processor from sensors locatedon the bracelet 105. In some instances, the information about theobjects located in the local environment 305 may also include depthinformation about one or more of the user's hands. The processor maydetermine if the hand gesture 330 refers to any of the objects detectedin the local environment 305. For example, the processor may determineif a hand gesture 330 indicates the user's interaction with the objects,such as if the hand gesture 330 is pointing at an object in the localenvironment 305. The processor thus interprets the user's hand gesture330 as determined by the processor from sensor information provided byany number of sensors located on the bracelet 105 within the context ofthe local environment 305 as determined by the imaging device 310. Insome examples, the processor uses information about the objects locatedin the local environment 305 to determine a hand gesture 330. Theprocessor may combine signals received from the bracelet 105 withinformation about objects received from the imaging device 310 todetermine a hand gesture 330. For example, if the sensors on thebracelet 105 indicate that the bracelet 105 is aligned with an objectdetected by the imaging device 3101, the processor may use thisinformation to determine that the user is pointing at the object, andthus that the hand gesture 330 is a pointing hand gesture.

In response to relating the hand gesture 330 to the detected objects,the processor may adjust the artificial reality provided to the userthrough any of the bracelet 105, eyewear device 110 and/or neckband 115.For example, if the processor determines that the hand gesture 330 ispointing at object 320, it may instruct the eyewear device 110 todisplay information to a user about object 320. For example, if object320 is a book that the hand gesture 330 is pointing at, then the eyeweardevice 110 may display to a user recent reviews about the book, books bythe same author, a preview of the book, etc.

Thus, by combining detection by the imaging device 310 and bracelet 105,the distributed artificial reality system is able to contextualize thehand gesture 330 and provide feedback to a user within the artificialreality environment in response to the hand gesture 330.

Signal Pathways in a Distributed System

FIG. 4 is a diagram of a signal pathway 400, in accordance with one ormore embodiments. In signal pathway 400, a signal source 410 sends afirst signal 415 to a mobile device 405. The mobile device sends asecond signal 420 to the neckband 115. The mobile device also sends athird signal 425 to a bracelet 105. The bracelet 105 sends a fourthsignal 430 to the neckband 115. This signal pathway may be reversed,such that the neckband 115 sends the fourth signal 430 to the bracelet105 and/or the bracelet 105 sends the third signal 425 to the mobiledevice 405. The connector 145 communicates a fifth signal 435 to theeyewear device 110. This signal pathway may be reversed, such that theeyewear device 110 sends a fifth signal 435 to the neckband 115 throughthe connector 145, the neckband 115 sends a second signal 420 to themobile device 405, and the mobile device sends a first signal 415 to thesignal source 410. First, second, third, fourth and fifth signals 415,420, 425, 430 and 435, respectively, may be transmitted and received inany order. They are described in FIG. 4 as “first,” “second,” “third,”“fourth,” and “fifth” for illustrative purposes only.

The signal source 410 is a wireless signal source, which may be capableof linking the mobile device 405 to the internet via the cellularnetwork of signal source 410. In some examples, the signal source 410 isa cellular network. The signal source may be from any combination oflocal area and/or wide area networks, using both wired and/or wirelesscommunication systems. The signal source 410 may use standardcommunication technologies and/or protocols to send first signal 415.Signal 415 may be sent using technologies such as Ethernet, 802.11,worldwide interoperability for microwave access (WiMAX), 3G, 4G, 5G,code division multiple access (CDMA), digital subscriber line (DSL),etc. In some examples, the mobile device may be any device havingcomputer functionality, such as a personal digital assistant (PDA), asmartphone, a laptop, tablet, or another suitable device. Mobile device405 is configured to communicate with the signal source 410, bracelet105 and the neckband 115. In some embodiments, the neckband 115 and/orbracelet 105 communicates directly with the signal source 410, such thatthe first signal 415 sent from the signal source 410 is sent directly tothe neckband 115 and/or bracelet 105.

The mobile device 405 communicates the second signal 420 to the neckband115 and the third signal 425 to the bracelet 105. The second signal 420and third signal 425 may be wired or wireless signals. The mobile device405 may have an application used to control the artificial realityenvironment produced by the bracelet 105, neckband 115 and/or eyeweardevice 110. The application may be run on any mobile device operatingsystem, such as an IOS operating system, an ANDROID operating system,etc. The application on the mobile device 605 may control other featureson the bracelet 105, neckband 115 and eyewear device 110, such asturning ON or OFF a voice command feature, adjusting volume, brightness,etc. The application on the mobile device 405 may allow for personalizedsettings of the bracelet 105, neckband 115 and eyewear device 110.

The mobile device 405 may serve as an additional processor to theprocessor located in the computation compartment of neckband 115.Through either of the second or third signals 420 and 425, the processorof the mobile device 405 is communicatively coupled to each of theneckband 115 and the bracelet 105. In some examples, the eyewear device110 may be communicatively coupled to the processor of the mobile device405 through the fifth and second signals 435 and 420. In other examples,the eyewear device 110 is communicatively coupled to the processor ofthe mobile device 405 through a direct signal (not shown) between theeyewear device 110 and the mobile device 405 through a wireless signal.The wireless signal between the eyewear device 110 and the mobile device405 may be a Bluetooth signal, Wi-Fi signal, or any other suitablewireless signal. The mobile device may process information from thebracelet 105, eyewear device 110 and/or the neckband 115. The mobiledevice 405 may have wired or wireless communication with neckband 115,depending on if there are latency issues with the receipt or transmittalof processing information. In some examples, the mobile device 405 mayserve as a battery backup for the battery compartments located on theneckband 115. In some examples, the neckband 115 receives informationabout sensors on the eyewear device 102 from the mobile device 405.

The second, third, fourth and fifth signals 420, 425, 430, 435,respectively, may be a wireless signal, such as a Bluetooth signal,Wi-Fi signal, or any other suitable wireless signal. In some examples,the second, third, fourth and fifth signals 420, 425, 430, 435,respectively, may be an electric, magnetic or optic signal conveyedthrough a wire, or any other non-wireless signal. The second signal 620may thus be information from the neckband 115 conveyed to the mobiledevice 405, or information from the mobile device 405 to the neckband115. This information may be processed by the mobile device 405.

The mobile device 405 is thus coupled to the neckband 115 by at leastthe second signal 420. A projector on the neckband 115 may project animage, video, or any visual content onto the screen of the mobile device405. The mobile device 405 may thus serve as an additional artificialreality device, onto which an artificial reality environment can beprojected from the neckband 115. The mobile device 405 may operate withthe bracelet 105 and/or neckband 115 only, such that the eyewear device102 is not included in the user's artificial reality experience and theartificial reality environment is entirely generated by the neckband 115and mobile device 405, or the bracelet 105 and the mobile device 405.

In some examples, the camera on the mobile device 405 may have a 3D usercapture feature, allowing the user's body to be 3-dimensionally renderedand tracked. The camera on the mobile device may provide a 3D depth mapof a user's face. The imaging device 310 may be located on the mobiledevice 405. In some examples, the mobile device 405 includes a separateimaging device that augments the information about the local environment305 as determined by the imaging device 310. For example, the mobiledevice 405 may have a passive camera or active camera including a depthcamera assembly, capable of scanning a room or otherwise mapping auser's environment. The camera assembly may be a TOF system, SL system,mono or stereo vision (passive and/or active), or any other systemcapable of producing a depth map. The camera assembly may provide afully 3D description of the user's environment, such as a room in whicha user is standing in or other physical structures around the user. Thecamera on the mobile device 405 may provide 3D information to thebracelet 105, neckband 115 and/or eyewear device 102.

The mobile device 405 may also have an audio and/or visual streamingfeature, allowing the user to perceive audio/visual information that maysupplement the artificial reality environment produced by the bracelet105, neckband 115 and/or eyewear device 102. The mobile device 405 mayinclude a haptic feature, wherein the user's physical interaction withthe mobile device 405 is translated as a command to the bracelet 105,neckband 115 and/or eyewear device 102.

The mobile device 405 may thus supplement the artificial realityenvironment provided by the bracelet 105, eyewear device 102 and/orneckband 115.

In addition to communicating through the mobile device 405, the bracelet105 may communicate directly to the neckband 115 through the fourthsignal 430. Fourth signal 430 may be a wireless signal, such as aBluetooth signal, Wi-Fi signal, or any other suitable wireless signal.In some examples, the fourth signal 430 may be an electric, magnetic oroptic signal conveyed through a wire, or any other non-wireless signal.The fourth signal 430 may thus be information from the neckband 115conveyed to the bracelet 105, or information from the bracelet 105 tothe neckband 115. This information may be processed by the neckband 115.For example, the bracelet 105 may communicate hand gesture 330 throughthe fourth signal 430 to the neckband 115.

Distributed Artificial Reality System

FIG. 5 is a block diagram of a distributed artificial reality system500, in accordance with one or more embodiments. The distributedartificial reality system 500 includes a bracelet 536, NED 502, aneckband 514, and a mobile device 534. The bracelet 536 is connected tothe NED 502, neckband 514 and the mobile device 534. The bracelet 536may be the bracelet 105 as described in FIGS. 1-4. The neckband 514 isconnected to the bracelet 536, the NED 502 and the mobile device 534.The neckband 514 may be the neckband 115 as described in FIGS. 1-2 and4. The NED 502 may be the eyewear device 110 as shown in FIGS. 1-2 and4. The mobile device 534 may be the mobile device 405 as shown in FIG.4. In alternative configurations of system 500, different and/oradditional components may be included. The system 500 may operate in aVR system environment, an AR system environment, an MR systemenvironment, or some combination thereof.

The NED 502 includes optical systems 135, as described with reference toFIG. 1. The NED 502 may also include an eye tracker 504, one or morepassive sensors 506, one or more active sensors 508, one or moreposition sensors 510, and an Inertial Measurement Unit (IMU) 512. Theeye tracker 504 may be an optional feature of the NED 502.

The eye tracker 504 tracks a user's eye movement. The eye tracker 504may include at least a dichroic mirror, for reflecting light from an eyearea towards a first position, and a camera at the position at which thelight is reflected for capturing images. Based on the detected eyemovement, the eye tracker 504 may communicate with the bracelet 536,neckband 514, CPU 520 and/or mobile device 534 for further processing.Eye tracking information collected by the eye tracker 504 and processedby the CPU 520 of the neckband 514 and/or mobile device 534 may be usedfor a variety of display and interaction applications. The variousapplications include, but are not limited to, providing user interfaces(e.g., gaze-based selection), attention estimation (e.g., for usersafety), gaze-contingent display modes (e.g., foveated rendering,varifocal optics, adaptive optical distortion correction, syntheticdepth of field rendering), metric scaling for depth and parallaxcorrection, etc. In some embodiments, a processor in the mobile device534 may also provide computation for the eye tracker 504.

Passive sensors 506 may be cameras. Passive sensors may also belocators, which are objects located in specific positions on the NED 502relative to one another and relative to a specific reference point onthe NED 502. A locator may be a corner cube reflector, a reflectivemarker, a type of light source that contrasts with an environment inwhich the NED 502 operates, or some combination thereof. In embodimentsin which the locators are active sensors 508 (i.e., an LED or other typeof light emitting device), the locators may emit light in the visibleband (˜370 nm to 750 nm), in the infrared (IR) band (˜750 nm to 1700nm), in the ultraviolet band (300 nm to 380 nm), some other portion ofthe electromagnetic spectrum, or some combination thereof.

Based on the one or more measurement signals from the one or moreposition sensors 510, the IMU 512 generates IMU tracking data indicatingan estimated position of the NED 502 relative to an initial position ofthe NED 502. For example, the position sensors 510 include multipleaccelerometers to measure translational motion (forward/back, up/down,left/right) and multiple gyroscopes to measure rotational motion (e.g.,pitch, yaw, and roll). In some embodiments, the IMU 512 rapidly samplesthe measurement signals and calculates the estimated position of the NED502 from the sampled data. For example, the IMU 512 integrates themeasurement signals received from the accelerometers over time toestimate a velocity vector and integrates the velocity vector over timeto determine an estimated position of a reference point of the NED 502.Alternatively, the IMU 512 provides the sampled measurement signals tothe neckband 514 and/or the mobile device 534 to process the computationrequired to estimate the velocity vector and the estimated position ofthe NED 502.

The IMU 512 may receive one or more calibration parameters from thebracelet 536, neckband 514 and/or the mobile device 534. The one or morecalibration parameters are used to maintain tracking of the NED 502.Based on a received calibration parameter, the IMU 512 may adjust one ormore IMU parameters (e.g., sample rate). The adjustment may bedetermined by the CPU 520 of the neckband 514, or a processor of themobile device 534. In some embodiments, certain calibration parameterscause the IMU 512 to update an initial position of the reference pointso it corresponds to a next calibrated position of the reference point.Updating the initial position of the reference point at the nextcalibrated position of the reference point helps reduce accumulatederror associated with the determined estimated position of the NED 502.The accumulated error, also referred to as drift error, causes theestimated position of the reference point to “drift” away from theactual position of the reference point over time. In some examples, theIMU 512 is located in the neckband 514 or an IMU is present in both theneckband 514 and NED 502. In some examples, the IMU 512 receivesposition information from both position sensors 510 on the NED 502,positions sensors 538 on the bracelet 536 and position sensors 510 onthe neckband (not shown).

As shown in FIG. 5, the neckband 514 includes an imaging device 310,power source 518, a CPU 520, a projector 526, user vitals monitor 528, awireless gateway 530, imaging device 310 and activator 532. The audiounit 516, projector 526, user vitals monitor 528, imaging device 310 andactivator 532 are optional components of the neckband 514. In someembodiments, the neckband 514 includes one or more multifunctionalcompartments that interface with various functional units. Thefunctional units can include, e.g., an additional power source, anadditional processing unit (e.g., CPU), the projector 526, the uservitals monitor 528, the wireless gateway 530, and the activator 532.

The imaging device 310 is optionally located on the neckband 514. Inother embodiments of the system 500, the imaging device 310 may belocated on the mobile device 534 or NED 502. The imaging device 310 isdescribed in further detail with reference to FIG. 3.

The power source 518 provides power to the optical systems 135, eyetracker 504, passive sensors 506, active sensors 508, position sensors510 and IMU 512. The power source 518 may be the battery compartment 155as shown in FIG. 1. Power source 518 may be a rechargeable battery,which may be recharged by the mobile device 534. The power source 518may be turned ON or OFF in response to an input of the activator 532,and/or a command received by the mobile device 534.

The CPU 520 may be any standard processor, and may be the processorembedded in the computation compartment 170 as shown in FIG. 1. The CPU520 may provide all computational processing for the bracelet 536 andthe NED 502, including the computation associated with the positionsensors 538, IMU 540, cameras 542, magnetic sensors 544, eye tracker504, passive sensors 506, active sensors 508, IMU 512. The CPU 520 maycarry out calculations in parallel with the processor of the mobiledevice 534. A processor in the mobile device 534 may provide calculationresults to the CPU 520.

The projector 526 may be located on the neckband 514 to project visualinformation to a user. The projector 526 may project visual informationonto a surface in the user's field of view, or onto a coupled devicewith a screen, such as the mobile device 534.

The user vitals monitor 528 monitors vital signs and other user healthindicators. Vital signs may be heart rate, pulse, estimated calorieconsumption, number of steps taken by the user, the user's temperature,respiration rate, blood pressure, etc. The user vitals monitor 528 maybe located in close proximity to a user's neck on the neckband 514, sothat the vital signs may be accurate. For example, a pulse detection ismore accurate if the vitals monitor is pressed firmly against the user'sneck. The user vitals monitor 528 may be thermally isolated from thepower source 518 and CPU 520 to ensure that temperature estimates are aresult of the user's temperature and are unaffected by heat generated bythe power source 518 and CPU 520. The user vitals monitor may be incommunication with the position sensors 510 and IMU 512 to detect usersteps and user movement to estimate the number of steps taken and/orcalorie consumption.

The wireless gateway 530 provides signal communication with the mobiledevice 534, bracelet 536 and/or the NED 502. The wireless gateway 530may convey the second signal 420 from the mobile device 405 to theneckband 115, as shown in FIG. 4. The wireless gateway 530 may conveythe third signal 425 as shown in FIG. 4. The wireless gateway 530 may beany standard wireless signal gateway, such as a Bluetooth gateway, Wi-Figateway, etc.

The activator 532 controls functions on the bracelet 536, neckband 514,the NED 502, and/or the mobile device 534. The activator 532 may powerON or OFF any of the units in the bracelet 536, NED 502 and/or neckband514.

The bracelet 536 includes position sensors 538, an IMU 540, cameras 542and magnetic sensors 544. The bracelet 536 may include any additionalpassive or active sensors. The position sensors 538 produce signals inresponse to movement of the bracelet 536. In some examples, the positionsensors 538 indicate a relative position between the bracelet 536 andany of the NED 502, neckband 514 and/or mobile device 534. The positionsensors 538 may be the position sensor 120 and are described in furtherdetail with respect to FIG. 1. The IMU 540 determines a position of thebracelet 536 using the signals produced by the position sensors 538. TheIMU 540 is also described with respect to FIG. 1. The cameras 542collect visual information about a user's hand to determine a user'shand gesture, such as hand gesture 330. The cameras 542 may be thecamera 125, and are described in further detail with respect to FIG. 1.The magnetic sensors 544 produce a signal indicating the direction of abackground magnetic field. Movement of the bracelet 536 produces achange in the relative position of the magnetic sensors 544 with respectto the magnetic field which can be used to determine a movement of thebracelet 105. Magnetic sensors 544 may be the magnetic sensors 130 andare described in further detail with respect to FIG. 1.

The distributed artificial reality system 500 produces an artificialreality environment to a user, or any combination thereof. Thedistributed artificial reality system 500 is able to distributeprocessing, power and heat generating functions across the bracelet 536,the NED 502, neckband 514 and mobile device 534. This allows each ofbracelet 536, the NED 502 and neckband 514 to be adjusted to the desiredweight and temperature for user comfort, as well as providing variedvirtual environment interfaces and functions for the user to interactwith at any of the bracelet 536, the NED 502, neckband 514 and/or mobiledevice 534.

Additional Configuration Information

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: measuring, by at least onemagnetic sensor on a bracelet device, movement of the bracelet devicerelative to a magnetic field of an environment; capturing, by at leastone camera on the bracelet device, one or more images of a hand of auser; determining, by at least one sensor on the bracelet device, aposition signal in response to movement of the hand; capturing, by adepth camera assembly (DCA), a plurality of optical signals from awaveform reflected off of at least one object in the environment, thewaveform emitted by the DCA into the environment; and wherein aprocessor, separate from and communicatively coupled to the braceletdevice and the DCA, determines gesture information describing a pose ofthe hand using a kinematic model and measurements from the at least onemagnetic sensor, the images, and the position signal, and wherein theprocessor determines a position of the hand in relation to theenvironment based in part on the gesture information, the positionsignal, and the plurality of optical signals.
 2. The method of claim 1,wherein the processor determines a location of the at least one objectin the environment based at least in part on the plurality of opticalsignals.
 3. The method of claim 2, wherein the processor determines theposition of the hand refers to the at least one object in theenvironment by comparing the location of the at least one object in theenvironment to the position of the hand.
 4. The method of claim 1,further comprising: presenting, by an eyewear device, content to theuser.
 5. The method of claim 4, wherein the content presented to theuser is based in part on the gesture information.
 6. The method of claim1, further comprising: performing a simultaneous localization andmapping calculation based in part on at least one of the measurementsfrom the at least one magnetic sensor, the images, and the positionsignal.
 7. The method of claim 1, wherein the kinematic model is amachine-learned model.
 8. The method of claim 1, further comprising:providing the measurements from the at least one magnetic sensor, theimages, the position signal, and the plurality of optical signals to theprocessor, wherein the processor is located in a mobile device that iscommunicatively coupled to the bracelet device.
 9. The method of claim1, wherein the processor is located in a neckband that iscommunicatively coupled to the bracelet device.
 10. The method of claim1, wherein the at least one sensor is selected from a group consistingof: a gyroscope, an accelerometer, and a magnetometer.
 11. The method ofclaim 1, further comprising: tracking, by at least one sensor on thebracelet device, the position of the hand over time based at least inpart on a previous position measurement.
 12. A non-transitorycomputer-readable medium storing instructions that, when executed by oneor more processors, cause the one or more processors to performoperations comprising: measuring, by at least one magnetic sensor on abracelet device, movement of the bracelet device relative to a magneticfield of an environment; capturing, by at least one camera on thebracelet device, one or more images of a hand of a user; determining, byat least one sensor on the bracelet device, a position signal inresponse to movement of the hand; capturing, by a depth camera assembly(DCA), a plurality of optical signals from a waveform reflected off ofat least one object in the environment, the waveform emitted by the DCAinto the environment; determining gesture information describing a poseof the hand using a kinematic model and measurements from the at leastone magnetic sensor, the images, and the position signal; anddetermining a position of the hand in relation to the environment basedin part on the gesture information, the position signal, and theplurality of optical signals.
 13. The non-transitory computer-readablestorage medium of claim 12, wherein the operations further comprise:determining a location of the at least one object in the environmentbased at least in part on the plurality of optical signals.
 14. Thenon-transitory computer-readable storage medium of claim 13, wherein theoperations further comprise: determining the position of the hand refersto the at least one object in the environment by comparing the locationof the at least one object in the environment to the position of thehand.
 15. The non-transitory computer-readable storage medium of claim12, wherein the operations further comprise: presenting, by an eyeweardevice, content to the user.
 16. The non-transitory computer-readablestorage medium of claim 15, wherein the content presented to the user isbased in part on the gesture information.
 17. The non-transitorycomputer-readable storage medium of claim 12, wherein the operationsfurther comprise: performing a simultaneous localization and mappingcalculation based in part on at least one of the measurements from theat least one magnetic sensor, the images, and the position signal. 18.The non-transitory computer-readable storage medium of claim 12, whereinthe kinematic model is a machine-learned model.
 19. The non-transitorycomputer-readable storage medium of claim 12, the operations furthercomprise: providing the measurements from the at least one magneticsensor, the images, the position signal, and the plurality of opticalsignals to a separate processor, wherein the separate processor islocated in a mobile device that is communicatively coupled to thebracelet device.
 20. The non-transitory computer-readable storage mediumof claim 12, the operations further comprise: tracking, by at least onesensor on the bracelet device, the position of the hand over time basedat least in part on a previous position measure.